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What this talk is 


An overview of some of the challenges and general principles when using machine learning in 
games 

Inspiration about what’s possible with Kinect 

Inspiration about what’s possible with machine learning and machine vision 




• What this talk isn’t 

• A step-by-step guide for writing machine vision systems 

• An academic breakdown of algorithms and code 
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Champion scanning in Kinect Sports Rivals 


• What is the Champion Scan? 

• Vision: “Awesome You” 

• Had to be You 

• Had to be Awesome 


Main problems: 

• Would it even work? 

• Making the experience playful yet accurate 

• Knowing that it was going to work for everyone 
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How The Scan Works 


• User moves into correct positions and p 
•We scan body & face 

• Classifiers determine facial features 

• Results used to assemble final character 





Tech made up of Machine Vision 


• Face Shape 

• Body Size 

• Glasses 

• Facial Hair 
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Classiriers 



Face Shape 


• Developed jointly by Rare and a team from Redmond 


• Approach is part of the Xbox One HDFace XDK 



• Available for developers to use 

• Also provides high-quality face animation tracking 

• 10% GPU and 1 Core CPU during shape computation 

• Much less (~12ms) for animation tracking 
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Face Shap 


e 


Then recursive PCA to extract blendshape parameters 

• http://en.wikipedia.org/wiki/Principal_component_analysis 




• 93 parameters (-sliders) in total 
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Face Shape 


• Overdrive the parameters to give more characterisation 


Apply parameters to “stylised” head 
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Key Learnings 



Optimise the right thing! 

• Avoid vague goals 

• Test your hypothesis 

• Optimise the correct metrics 


You need lots of data 

• A small amount of data leads to false confidence 
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First we measure height using Kinect skeleton tracking 
Then torso width and extent using depth and RGB feed 
Then apply to final model 


m 
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Key Learnings 


• Hard to get positive aspirational result 

• Weight often key part of visual identity 

• Weight not often “aspirational” 



Solution was to find an aesthetic which validates size, but not unflattering 
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Raw classifier developed by team in Redmond 
Available as part of Xbox One Expressions XDK 
Uses ActiveIR for lighting independence 

Random Decision Forest classifier, trained with thousands of 
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Expressions API gives us a point result 

• Noisy 

• We need to average the result over frames and test above a tolerance 

• Initially tweaked by hand, then automated in a script 


Not a binary classification 

Created a “low confidence” beard to cope with false positives 
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Machine learning does not have to be complex! 

e as a brute force offline tool 


as sun 


Weight your failure cases to get best results 

• Score “correct” results highest 

• Then “acceptable” 

• Then “ok” 

• And sort results to eive the best overall result 
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Your brain is very good at estimating lighting models 


Computers are not 


#787878 
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important is 
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Key Learnings 


• You need to manage expectations with Machine Learning systems. 

• Perception was that it was “easy” problem 

• Unlikely that any Machine Learning system will hit 100% 



• Identify problem data for your approach, and source more 
of it! 


•We gathered lots of clips of people in low-light conditions 

• Allowed us to quickly test hypotheses to see if they showed promise 

• Iterate, Iterate, Iterate! 
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Hair Style 
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Uses combination of depth feed & hair segmentation in RGB 
Estimates volumes of hair for: Top, Side & Below Ear 
Picks most appropriate hair asset based on results 
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If subject is male and has very short hair we run a bespoke baldness 
classifier 


Looks for curvature of the forehead 


• Looks to find hairline using; RGB feed 
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Fringes & Pony Tails 



Use hair segmentation for fringe 



If subject is female and has short hair, we assume a pony tail 






Hair Colour 



Average RGB pixels from hair 
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Key Learnings 



If we don’t detect long hair on females we assumed a pony tail 
Least offensive “wrong” result 


If a result is sensitive add las many layers of security as possible 

• 3 separate tests for baldness 

• Very low false positive result 
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Animation 







• The final result needed to be deformable, yet animate 

4P J 

• In total 490 blendshapes to deform and animate head 

• Full animation rig mapped onto blendshapes 



GPU bottleneck was transferring blendshapes to GPU 

• Optimisation was to bake “static” blendshapes into new mesh 

• Only transfer animation blendshapes 
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How We Validated Our Results 


We sourced 850 clips of people being scanned. 

• 6 territories (London, Madrid, Turkey, Japan, China, US) 

• Strategic mixture of age, gender & ethnicity 
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How We Validated Our Results 


Each clip annotated to give “ground truth” details about subject. 

• Simple csv file with id, path to recording and results expected 


Iclip 

l\\rareshar 
l\\rareshar 
l\\rareshar 
l\\rareshar 
l\\rareshar 
i\\rareshar 
Wrareshar 
|\\rareshar 
l\\rareshar 
l\\rareshar 
l\\rareshar 
l\\rareshar 
l\\rareshar 


age 


gender exp_facialhair expjjlasses exp_haircol 


exp_haircategory exp_fringe 


adult 

male 

yes 

no 

brown 

short 

yes| no 

light 

adult 

male 

yes no 

no 

grey 

short 

no 

light 

adult 

male 

no 

no 

blonde | brown 

short | buzz 

no 

light 

adult 

male 

no 

no 

brown 

short 

no 

light 

adult 

male 

no 

yes 

black. 

short 

no 

light 

adult 

male 

no 

no 

blonde 

short 

yes no 

light 

adult 

male 

yes | no 

yes 

bald 

bald 

no 

light 

adult 

female 

no 

yes 

brown 

pony 

no 

light 

adult 

male 

yes | no 

no 

brown 

short: | tall 

no 

light 

adult 

female 

no 

no 

black 

long 

no 

light| mec 

adult 

male 

yes 

no 

black 

short | tall 

no 

light 

adult 

male 

yes | no 

no 

black 

short 

no 

light 

adult 

male 

yes 

no 

black 

buzz 

no 

light 


exp_skin exp_body 

thin | medium 
thin | medium 
thin | medium 
large 

thin | medium 
thin | medium 
medium | large 
medium | large 
thin I medium 


thin | medium 
thin | medium 
thin I medium 
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How We Validated Our Results 


Automatic process to run each clip with latest code 

• Hooked into our automatic build process 


Ran on 16 devkits in 3 hours 


Twice daily 






+ ADULT i'-fALES CORRECT 


+ HAIR LENGTH CORRECT 


+ BALDNESS CORRECT 


+ FRINGE CORRECT 


+ LIGHT 5 KIIN CORRECT 


+ MEDIUM SKIN CORRECT 


+ DARK 5KIN CORRECT 


30 . 5 % ( 57 / 63 ) 


SKIIN COLOUR PICKER 


33 . 7 % ( 605 / 607 ) 












01/08/2013 


21/08/2013 


1Q/D9/2013 


30 / 08/2013 


09/11/2D13 


29/11/M13 


19 / 12/2013 


08/01/2014 


23/13/2313 


SfcnCdcir 


Hair Colour 


’&8dncs 


How We Validated Our Results 


over time 
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Key Learnings 


Machine learning lives or dies on the quality of source data 


• 24 hour cycle of improve, observe, validate, repeat 



• Cut corners where you can 

• You are unlikely to hit 100%, so goal is to maximise results 

• Test a simple assumption, it might save a lot of work 
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Some Results 








Some Results 




Some Results 



MADRID- SUBJECTl 33 
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Some Results 








Some Results 







Some Results 







Scanning Experience 


Whimsical/playful tone 


• Dr Who! 


• Required a LOT of User Research 

• Biggest challenge: positioning the user 
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Scanning Experience 

• 24 hour cycle of User Research and reaction 

• All engineers observed sessions 

• Quick deadline to verify improvements 


AUGUST 11-13,2014 



PERS CONFERENCE" EUROPE 2014 





Scanning Experience 


• The reveal cutscene 

• Create tension and anticipation 

• Fun payoff to the experience 



What Went Well? 


• End result is good 

• Scanning flow works well for almost all users 

• Machine vision works well for most users 


• Automated testing gave us launch confidence 



Gf4E OE^fcOPERS CONFERENCE" EUROPE 2014 AUGUST 11-13, 2014 



GDCEUROPE.OJM 


What Could We Have Done Better? 


• Data Capture was started late. 
• Get data early! 



• “Experience” user research was started early enou 
due to missing build functionality 

• Result trends towards generic for -50% of users 

• Hair styles were correct, but often uninspiring 
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uestions? 



ndy.bastable@microsoft.com 


#GDCEurope 
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